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Introduction 


The  Department  of  Defense  (DoD)  spends  at  least  60%  of  its  budget  for  the 
acquisition  and  sustainment  of  new  material  capability  and  services.  The  DoD’s  track 
record  with  regard  to  maximizing  value  from  this  investment  has  been  poor,  to  say 
the  least,  and  warrants  a  more  in-depth  understanding  of  the  root  cause  of  the 
problem.  The  lack  of  a  successful  integrated  network  approach  to  the  problem  of 
poor  program  performance  is  evident  in  the  many  significant  cost  and  schedule 
program  overruns  and  the  failure  of  legislation  and  policy  emanating  from  initiatives 
such  as  the  Packard  Commission  study,  Goldwater-Nichols  Legislation,  and  more 
recently,  the  Better  Buying  Power  initiative. 

While  the  DoD  acquisition  framework  is  well  defined  and  incorporates  a 
variety  of  disciplines  such  as  systems  engineering,  budgeting,  contracting,  test  and 
evaluation,  and  so  forth,  it  does  not  address  the  fundamental  variable  that  those 
responsible  for  administering  the  process  do  not  necessarily  behave  in  linear  and 
predictable  ways.  The  current  acquisition  decision  support  system  of  requirements 
development,  budgeting,  and  acquisition  management  (Figure  1)  presumes  that  “fact 
of  life”  variation  throughout  the  life  of  the  program  can  be  managed  in  a  predictable 
way  and  that  the  acquisition  workforce  behaves  under  the  Homo  economicus,  or 
economic  man,  principle.  The  economic  man  is  the  concept  in  many  economic 
theories  which  portrays  humans  as  consistently  rational  and  narrowly  self-interested 
agents  who  usually  pursue  their  subjectively-defined  ends  optimally. 
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Figure  1.  Defense  Acquisition  Decision  Support  System  (DSS)  (DAG,  2011) 
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The  acquisition  process  within  the  DSS  and  described  in  DoD  5000.02, 
consists  of  a  multi-phase  process  designed  to  integrate  systems  engineering, 
budgeting,  and  management  across  the  program  through  time  (Figure  2).  Within 
each  of  the  phases  are  key  decision  and  knowledge  points  at  which  decision-makers 
assess  the  overall  performance  of  the  program  relative  to  the  key  program  attributes 
of  cost,  schedule,  and  performance. 
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Figure  2.  Defense  Acquisition  Process  (DoD  5000.02,  2015) 

Reforms  to  the  defense  acquisition  process  have  focused  on  the 
methodological  procedures  of  the  acquisition  process  providing  only  partial  remedies 
because  they  do  not  address  incentives  to  deviate  from  defined  practices.  Weapons 
acquisition  is  a  complicated  enterprise,  complete  with  unintended  incentives  that  are 
motivated  by  the  perceived  and  real  circumstances  within  which  the  program  is 
performing.  These  incentives  stem  from  several  factors.  For  example,  the  different 
participants  in  the  acquisition  process  impose  conflicting  demands  on  weapon 
programs  so  that  their  purpose  transcends  just  filling  voids  in  military  capability.  The 
budget  process  forces  funding  decisions  to  be  made  well  in  advance  of  program 
decisions,  which  encourages  undue  optimism  about  program  risks  and  costs. 

Finally,  program  office  personnel  respond  to  the  changing  environment  based  upon 
their  intrinsic  understanding  of  the  business  conditions  and  their  level  of  experience 
and  comfort  in  this  complex  procurement  environment. 

Concept  and  Scope 

This  proposed  research  experiment  will  examine  the  decision-making  process 
within  the  program  office  and  the  self-organization  of  key  program  office  personnel 
based  upon  formal  and  informal  communications  links.  Additionally,  we  are 
interested  in  the  effects  of  this  self-organizing  process  on  the  organization’s  shared 
situational  awareness  and  ultimately  how  these  decisions  evolve  and  impact  overall 
program  performance. 

This  experiment  will  help  us  determine  the  impact  of  individual  relationships 
within  a  program  office  and  how  their  decision-making  and  task  execution  impacts 
the  overall  performance  of  a  program.  We  will  examine  these  relationships  by 
observing  the  communications  network  within  the  organization  and  the  various 
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Disposal 


formal  and  informal  ties  that  manifest  themselves  as  a  program  progresses  through 
the  acquisition  process. 


The  focus  of  this  experiment  will  be  an  Acquisition  Category  1 D  (ACAT 1 D) 
program.  We  will  use  the  Airborne  Joint  Tactical  Radio  System  (AMF)  program  as 
our  program  of  reference  due  to  the  extensive  data  available  for  this  program.  The 
descriptive  model,  which  is  a  process  that  describes  real-world  events  and  the 
relationships  between  factors  responsible  for  them,  for  the  self-organizing 
communications  process  relationship  is  shown  in  Figure  3.  There  are  10  attributes  in 
this  descriptive  model,  which  are  considered  intervening  variables.  An  intervening 
variable  is  an  internal  state  that  is  used  to  explain  relationships  between  observed 
variables,  such  as  independent  and  dependent  variables.  With  the  exception  of  the 
formal  and  informal  communications  pathways,  we  will  hold  all  of  the  attributes 
constant  throughout  the  experiment. 


Figure  3.  Self-Organizing  Descriptive  Model  for  a  Defense  Acquisition  Program  Office 
Nousmaking 

I  refer  to  the  process  of  individual  situational  awareness  (SA)  as 
Nousmaking,  which  is  the  degree  to  which  the  decision  maker  optimizes  four 
aggregate  categories;  explicit  knowledge,  tacit  knowledge,  trust,  and  sensemaking 
and  is  able  to  make  high  quality  decisions  in  ambiguous  environments.  Nousmaking 
comes  from  the  term  Nous,  which  means  to  have  intelligence  and  have  the  ability  to 
make  good  judgments  and  decisions  (Webster,  2016).  In  essence,  the  individual  is 
determining  what  is  real  through  the  interaction  of  the  four  aggregate  categories. 
These  aggregate  categories  were  derived  from  previous  research  conducted  on 
Special  Operations  soldiers’  decision-making  process  in  complex  and  volatile 
conditions.  The  results  of  this  research  are  documented  in  The  Creation  of  Nous 
through  the  Interaction  of  Sensemaking,  Trust,  Tacit  Knowledge  and  Explicit 
Knowledge  and  its  Relationship  to  Decision  Making  in  Complex  and  Chaotic 
Environments  (Jones,  2015).  The  degree  to  which  the  individual  optimizes  the 
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aggregate  categories  and  is  able  to  make  timely  and  high  quality  decision 
determines  the  overall  effectiveness  of  the  decision.  While  this  is  a  significant 
attribute  in  the  descriptive  model  with  regard  to  understanding  the  human  impact  on 
the  acquisition  process,  we  will  simply  measure  the  observable  communication 
patterns,  as  a  result  of  the  independent  variable  changes  during  this  experiment  and 
leave  the  relationship  of  Nousmaking  on  the  overall  performance  to  future  research. 
The  relationship  to  the  network  communications  structure  and  the  descriptive  model 
is  shown  in  figure  4. 


Figure  4.  Self-Organizing  Network  Communications  Pattern 


(Note:  Network  structure  on  the  right  depicts  all  the  potential 
communications  pathways  in  an  organizational  structure) 


Self-Organizing  Network  Behavior 


The  nodes  in  the  network  are  specific  individuals  within  well-defined 
organizational  positions.  Individuals  must  meet  certain  pre-requisites  to  be  qualified 
for  their  respective  position.  These  pre-requisites  are  based  on  the  Defense 
Acquisition  Workforce  Improvement  Act  (DAWIA)  legislation,  which  mandated  a 
certain  certification  level  for  critical  acquisition  positions.  For  the  purposes  of  this 
experiment,  we  will  use  actual  program  office  personnel  with  the  appropriate  DAWIA 
certification  levels.  The  strength  and  manner  in  which  communications  manifest 
themselves  within  the  program  office  results  in  the  self-organizing  effect,  which 
ultimately  impacts  the  program  decision  strategy  (de  Montjoye,  Stopczynski, 
Shmueli,  Pentland,  &  Lehmann,  2014). 

There  are  fundamentally  three  communications  links  we  will  examine  for  this 
experiment:  expressive,  instrumental,  and  structural  (Table  1).  These  networking 
relationships  are  based  upon  research  by  de  Montjoye  et  al. ,  in  which  performance 
was  strongly  correlated  with  the  network  of  expressive  and  instrumental  ties.  These 
three  communications  relationships  form  the  basis  of  the  dynamic  network  of  shared 
understanding  within  the  program. 
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Table  1.  Communications  Network  Ties 


Ties 

Definition 

Instrumental 

Ties  which  arise  in  professional  settings,  between  colleagues 
interacting  and  spending  time  together. 

Expressive 

Ties  that  reflect  friendship  or  personal  relationship  and  include  an 
affective  factor. 

Structural 

Ties  which  are  predictable  based  upon  the  pre-defined 
organizational  communications  structure. 

Data  will  be  collected  to  measure  the  strength  of  the  ties  within  the  formal  and 
informal  communications  network  as  the  program  proceeds  through  a  time-phased 
scenario  that  simulates  the  technology  development  phase  of  the  program.  Figure  5 
shows  the  formal  organizational  command  and  control  structure  and  linear 
communications  pathways  relative  to  all  possible  communications  network  ties 
within  a  program  office. 
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Figure  5.  Organizational  Structure  Relative  with  Self-Organizing  Network  Structure 

The  program  communications  and  decision-making  network  structure  reflects 
the  links  that  develop  as  a  result  of  the  self-organizing  process  within  the  program 
office  and  ultimately  informs  the  program  decision  strategy. 

Participants  for  this  experiment  will  include  Level  III  certified  government 
acquisition  professionals  and  industry  representatives.  Eventually,  this  research  will 
involve  the  following  five  phases  of  the  experimental  process: 

■  Phase  1 :  Definition  and  experimental  design 
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■  Phase  2:  Discovery  experiment  execution  involving  Level  III  certified 
acquisition  professionals  representing  critical  program  office  acquisition 
positions 

■  Phase  3:  Preliminary  hypothesis  testing  to  examine  decision  and  behavior 
pattern  impacts  of  changing  predetermined  independent  variables 

■  Phase  4:  Refined  hypothesis  testing  to  examine  how  decision  and 
behavior  variation  will  support  organizational  and  process  changes  to 
improve  program  value  and  return  on  investment 

■  Phase  5:  Conduct  demonstration  to  validate  findings  and  potential  for 
future  modeling  to  support  improved  acquisition  decision-making 

This  paper  addresses  the  first  two  in  that  the  initial  experiment  will  be  a 
discovery  type  experiment,  which  will  lead  to  the  formulation  of  an  initial  hypothesis 
for  follow-on  experimentation.  This  experiment  will  be  aligned  around  the  following 
question:  How  do  the  communications  and  information  exchange  patterns  within  a 
program  office  impact  the  overall  program  performance  outcomes.  More  specifically, 
given  that  there  are  formally  established  communications  requirements  (ties)  within 
a  program  office,  how  do  these  communications  ties  vary  and  influence  the 
decisions  within  an  ACAT  ID  program  office  chartered  to  develop  a  highly  complex 
networking  radio  system? 

Literature  Review 

This  literature  review  will  provide  a  summary  of  relevant  research  within  the 
field  of  decision-making  and  network  theory  to  help  us  better  understand  the  relevant 
nature  of  the  individual  impact  on  organizational  outcomes.  We  will  look  at  how 
string  ties  between  individuals  impact  the  collaborative  problem  solving  process, 
systems  thinking  and  complexity,  and  self-organizing  behavior  in  an  organization. 

Strong  Ties  in  Collaborative  Problem  Solving 

There  is  a  growing  body  of  evidence  to  suggest  that  an  organization’s 
performance  is  related  to  the  strength  of  the  ties  between  individuals  and  teams 
through  the  formal  and  informal  information  networks  within  an  organization.  De 
Montjoye  et  al.  (2014)  suggested  that  a  team’s  performance  is  strongly  correlated 
with  its  networks  of  expressive  and  instrumental  ties  and  that  only  the  strong  ties  will 
have  an  appreciable  impact  on  the  overall  performance.  They  conducted  a 
qualitative  and  quantitative  study  on  the  performance  of  45  teams  looking  at  the 
performance  outcomes  of  these  teams  relative  to  their  information  network  ties. 
Previous  understanding  of  these  relationships  has  been  difficult  to  quantify  in  that 
the  understanding  of  the  relationship  between  social  connectivity,  information  flow, 
and  team  performance  has  existed  in  the  form  of  advice,  expertise,  and  implicit  and 
tacit  knowledge.  The  research  of  de  Montjoye  et  al.  (2014)  built  upon  the  advances 
in  social  network  analysis,  showing  the  impact  of  an  individual’s  information  or 
collaboration  network  on  their  performance.  Essentially,  they  concluded  that  the 
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structure  of  social  interactions  can  enhance  or  hinder  access  to  relevant  information, 
thus  impacting  overall  performance.  Additionally,  they  bridged  the  literature  on  team 
performance  and  information  networks,  leading  them  to  the  conclusions  that  only  the 
strong  ties  within  these  networks  explained  more  of  the  performance  variance  than 
any  other  factor  within  the  teams. 

Their  work  is  significant  and  directly  relates  to  the  proposed  experiment  in 
that  we  are  exploring  the  impact  of  internal  communications  networks  within  a 
defense  program  organization  and  the  effects  of  these  network  ties  on  the  overall 
performance  of  the  program.  Understanding  that  there  are  varying  levels  of 
communications  networks  within  a  program  office  and  that  these  networks  could 
have  an  impact  on  program  performance  will  help  us  to  establish  potential  cause 
and  effect  based  upon  the  strength  and  centrality  of  these  network  connections 
relative  to  the  formal  structural  patterns  of  expected  communications. 

Systems  Thinking  and  Complexity 

Systems  engineering  provides  proven  methodologies  to  analyze  and  define 
the  management  function.  In  fact,  as  an  analytical  process,  systems  engineering 
decomposes  system  problems  into  component  parts  to  provide  for  an  optimal 
solution.  In  the  case  of  program  business  functions,  these  analytical  steps  include  a 
quantitative  evaluation  of  the  relationships  and  interactions  among  and  between  the 
key  variables  in  the  program  office,  manpower,  information  systems,  and 
stakeholders  and  their  interdependencies. 

The  systems  engineering  principle  of  decomposition  provides  a 
methodological  process  to  not  only  identify,  but  also  measure  the  inputs,  the  time 
and  cost  associated  with  the  process  itself,  and  the  outputs.  For  the  same  reason 
systems  engineering  uses  requirements  traceability  to  ensure  adherence  to  system 
requirements,  the  analytical  process  provides  a  means  of  comparing  business 
process  outputs  to  the  inputs,  as  well  as  measuring  those  outputs  in  terms  of 
efficiencies  and  effectiveness. 

Systems  engineering  supports  the  development  and  maintenance  of  good 
design.  That  design  leads  to  decisions  in  weapon  systems  development.  In  this 
analysis  of  program  office  business  processes  and  network  communications 
structures,  we  should  be  able  to  identify  either  an  improved  design  for  the  flow  of 
information  or  a  self-organizing  social  network  that  reflects  the  positive  or  negative 
effects  of  the  program  environment.  The  result  of  this  analysis  could  be  an  improved 
design  for  the  flow  of  information  within  the  management  function  of  the  Program 
Management  Organization  (PMO).  The  emphasis  of  the  management  work  needs  to 
be  on  the  management  system,  rather  than  the  piece  parts  and  daily  responses 
typical  of  the  PMO  workday. 

Systems  thinking  and  its  application  to  management  have  received  great 
attention.  Early  studies  emphasized  the  importance  of  defining  management  as  a 
systems  activity  (Jenkins  &  Youle,  1968;  Johnson,  Kast,  &  Rosenzweig,  1964; 
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Sterman,  1996).  More  recently,  the  continued  development  of  systems  engineering 
as  discipline  has  fostered  a  renewed  interest  in  applying  systems  thinking  and 
systems  engineering  principles  to  management  problems  (Checkland,  1994;  Sage  & 
Rouse,  2009).  A  systems  approach  to  project  management  would  complement  the 
increased  emphasis  of  systems  engineering  and  weapon  systems  development.  Key 
to  this  idea  is  that  system  engineering  management  of  the  technical  aspects  of 
development  should  be  mirrored  by  a  systems  approach  in  the  management  of  that 
technical  effort  (Feigenbaum  &  Sasieni,  1968). 

A  constant  theme  in  the  management  science  literature  is  the  criticality  of 
addressing  project  complexity.  It  is  important  to  recognize  that  managerial  and 
technical  complexity,  coupled  with  the  limits  of  human  capability,  has  resulted  in 
managerial  and  technical  specialization.  The  specialists  are  experts  in  their 
particular  field,  but  that  local,  limited  knowledge  of  the  field  precludes  identifying 
potentially  optimal  solutions  to  interdependent  program  problems  (Amaral  &  Uzzi, 
2007).  Specialization  has  a  limiting  function,  in  that  the  specialists  in  a  PMO  are 
measured  by,  and  capable  of,  addressing  only  those  issues  in  their  specific  area. 
This  suggests  that  requests  for  information  or  expertise  outside  a  specialist’s  area 
may  have  a  debilitating  effect  on  the  efficiency  and  effectiveness  of  the  PMO. 

Self-Organizing  Systems 

While  defense  program  organizations  have  evolved  into  functionally  aligned 
processes  with  certified  experts  in  each  of  these  discrete  functional  domains,  it  is 
precisely  this  phenomenon  that  creates  the  potential  for  an  organization  to  seek  a 
functional  decision-making  equilibrium  beyond  the  prescribed  formal  organizational 
structure.  Waldrop  (1992)  suggests  that  balancing  between  rigidity  and  turbulence  is 
necessary  for  adaptation  and  self-organization.  This  is  commonly  referred  to  as  “the 
edge  of  chaos.”  According  to  Heylighen  (2008),  complex  systems  are  modeled  as 
agents  and  act  upon  their  environment  in  response  to  the  events  they  experience. 
Additionally,  these  agents  are  assumed  to  be  goal-directed  and  their  actions  tend  to 
maximize  their  individual  “fitness”  or  preferences. 

Defense  program  management  organizations  are  structured  to  meet  the  cost, 
schedule,  and  performance  objectives  of  their  respective  technology  requirements. 
The  processes  they  follow  are  well  defined  in  both  statute  and  regulations,  and  all 
individuals  are  “trained”  in  their  respective  functional  areas  and  generally  have  some 
level  of  experience  in  these  domains.  These  organizations  are  generally  rigid  in 
nature  with  very  little  variation  from  doctrinal  design.  This  experiment  will  examine 
the  extent  to  which  a  defense  program  office  self-organizes  as  a  program  becomes 
more  turbulent  and  how  the  complex  network  of  social  interaction  and  decision¬ 
making  changes  from  its  perceived  base  state  as  defined  by  regulation  and  law. 
Self-organization  is  defined  as  the  spontaneous  emergence  of  global  structure  out  of 
local  interactions  (Heylighen,  2008).  The  spontaneous  nature  of  an  organization 
suggests  that  there  is  no  internal  or  external  control  over  the  process;  rather,  it  is 
being  shaped  by  individuals  reacting  to  the  environment  and  potentially  changing  the 
inherent  network  structure  within  an  organization. 
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Ashby  (1962)  refers  to  this  phenomenon  as  a  system  changing  itself  from  a 
“bad”  way  of  behaving  to  a  “good”  way  of  behaving.  Essentially,  as  a  program 
organization  becomes  less  productive,  it  will  self-organize  in  an  effort  to  become 
more  productive.  Given  the  opportunity  to  self-organize  to  an  optimal  state, 
unfettered  by  organizational  inertia,  the  organization  should  seek  equilibrium 
between  its  original  rigid  state  and  a  chaotic  state,  thus  achieving  a  higher  degree  of 
performance. 

The  work  of  Levitt,  Thomsen,  Christiansen,  Kunz,  Jin,  and  Nass  (1999)  explore, 
extend  and  operationalize  Galbraith’s  (1973)  information  processing  view  of 
organizations.  Levitt  et  al.  studied  project  work  processes,  organizational  micro¬ 
contingency  theory,  and  organizational  design  using  Virtual  Design  Teams  (VDTs). 
These  VDTs  simulated  the  micro-level  information  processing,  communications,  and 
coordination  behavior  of  participants,  predicting  participant  and  project-level 
performance.  Additionally,  follow-on  simulations  included  measures  of  activity 
flexibility,  complexity,  uncertainty,  and  interdependence  strength  between  nodal 
relationships.  Their  simulations  sought  to  model  more  flexible  organizations 
executing  less  routine  processes,  bridging  the  gap  between  cognitive  and  social 
psychological  micro-organization  theory  and  sociological  and  economic  macro- 
organizational  theory  for  project  teams.  This  work  will  provide  a  reference  by  which 
the  dynamic  nature  of  a  PMO  changes  based  upon  environmental  inputs,  and  the 
interrelationship  with  the  social  bureaucratic  influences. 

Experimental  Research  Design 

The  Joint  Tactical  Radio  System  (JTRS)  is  a  family  of  affordable,  high- 
capacity,  programmable,  multi-band/multi-mode  tactical  radios  designed  to  provide 
both  line-of-sight  and  beyond-line-of-sight  communications  capabilities  to 
warfighters.  JTRS  uses  software-defined  radio  technology  to  achieve  the  needed 
flexibility,  upgradeability,  and  interoperability.  JTRS  is  an  enabler  for  conducting  joint 
operations  with  full  spectrum  dominance.  The  goal  of  the  JTRS  program  is  to 
develop  a  radio  system  that  satisfies  the  requirements  stated  in  the  Capabilities 
Development  Document  (CDD)  approved  by  the  Joint  Requirements  Oversight 
Council  (JROC).  This  document  calls  for  the  radio  system  to  be  “software- 
reprogrammable,  multi-band/multi-mode  capable,  networkable,  and  provide 
simultaneous  voice,  data  and  video  communications  with  low  probability  of  intercept. 
These  requirements  will  ensure  that  the  radio  system  will  be  able  to  interoperate  with 
legacy  radios  using  their  existing  waveforms,  a  concept  known  as  backward 
compatibility.  Figure  6  shows  the  operational  view  for  the  JTRS  program. 

JTRS  is  organized  into  five  ACAT  1 D  program  offices,  each  with  the 
requirement  to  develop  a  networking  system  for  different  parts  of  the  overall  network 
air,  ground,  and  sea  battlespace.  We  will  focus  on  one  of  these  ACAT  ID 
programs — the  Airborne,  Maritime,  Fixed  Station  Program. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


■9- 


Parameter  Space  Identification 

To  better  understand  the  influence  of  the  communications  network  on  the 
program  decision-making  strategy,  we  must  identify  the  parameters  within  which  we 
will  assess  our  fundamental  question  and  later  develop  a  refined  hypothesis.  The 
three  primary  groups  we  are  interested  in  defining  are  the  dependent,  independent, 
and  intervening  variables.  These  can  be  described  as  design  variables,  criteria 
space,  and  functional  constraints,  respectively,  within  the  parameter  space 
identification  (PSI)  framework.  Table  2  shows  the  variables  for  this  experiment  within 
the  context  of  the  PSI  framework. 

Table  2.  Experimental  Variable  Within  the  PSI  Framework 


Design  Variables 

(Independent  Variables) 

Description 

Budget  Variance 

Program  budget  will  be  changed  at  specific  points  in  the  timeline  to 
assess  the  impact  on  the  criteria  space. 

Time  Variance 

Program  time  constraints  will  be  changed  at  specific  points  in  the 
timeline  to  assess  the  impact  on  the  criteria  space. 

External  Program  Threats 

External  threats  are  the  levels  of  program  advocacy  by  program 
sponsors  and  stakeholders  over  time.  The  range  of  program  advocacy 
variation  that  will  be  scored  on  a  scale  of  1-10. 

Functional  Constraints 

(Intervening  Variables) 

Instrumental 

Communications  Link 

These  are  ties  which  usually  arise  in  professional  settings,  between 
colleagues  interacting  and  spending  time  together.  (Instrumental  ties 
will  be  reflected  by  the  color  green  to  show  the  central  tendency  of 
instrumental  ties  within  the  organization  during  key  decision  periods.) 

Expressive  Communications 
Link 

These  are  ties  that  reflect  friendship  or  personal  relationship  and 
include  an  affective  factor.  (Expressive  ties  will  be  reflected  by  the 
color  red  to  show  the  central  tendency  of  instrumental  ties  within  the 
organization  during  key  decision  periods.) 

Structural  Communications 
Link 

These  ties  are  predictable  based  upon  the  pre-defined  organizational 
command  and  control  structure.  (Structural  ties  will  be  reflected  by  the 
color  black  to  show  the  central  tendency  of  structural  ties  within  the 
organization  during  periods  of  key  decision-making.) 

Criteria  Space  (Dependent 
Variable) 

Program  Cost 

Program  cost  will  be  measured  by  the  amount  of  variation  from  the 
program  performance  measurement  baseline  at  the  start  of  the 
program. 

Program  Schedule 

Program  schedule  will  be  measured  by  the  amount  of  variation  from 
the  program  performance  measurement  baseline  at  the  start  of  the 
program.  Earned  value  management  data  will  be  used  to  determine 
program  schedule  variance. 

Program  Performance 

Program  performance  will  be  measured  by  the  amount  of  variation 
from  the  program  Key  Performance  Parameters  (KPPs)  as  a  function 
of  time.  The  measures  of  effectiveness  (MOEs)  will  be  based  upon 
predetermined  technology  readiness  levels,  defined  in  Table  3. 

Technology  readiness  levels  (TRLs),  as  defined  in  the  201 1  DoD  Technology 
Readiness  Assessment  Guidance,  are  a  method  of  estimating  technology  maturity  of 
critical  technology  elements  (CTEs)  of  a  program  during  the  acquisition  process. 
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They  are  determined  during  a  Technology  Readiness  Assessment  (TRA)  that 
examines  program  concepts,  technology  requirements,  and  demonstrated 
technology  capabilities.  TRLs  are  based  on  a  scale  from  1  to  9,  with  9  being  the 
most  mature  technology.  The  use  of  TRLs  enables  consistent,  uniform  discussions 
of  technical  maturity  across  different  types  of  technology. 


Table  3.  Technology  Readiness  Level  (TRL)  Definitions 


Description 

TRL  1. 

Basic  principles  observed  and  reported 

TRL  2. 

Technology  concept  and/or  application  formulated 

TRL  3. 

Analytical  &  experimental  critical  function  and/or 
characteristic  proof-of-concept 

TRL  4. 

Component  and/or  breadboard  validation  in  laboratory 
environment 

TRL  5. 

Component  and/or  breadboard  validation  in  relevant 
environment 

TRL  6. 

System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment  (ground  or  space) 

TRL  7. 

System  prototype  demonstration  in  a  space  environment 

TRL  8. 

Actual  system  completed  and  “Flight  qualified”  through  test 
and  demonstration  (ground  or  space) 

TRL  9. 

Actual  system  “Flight  proven”  through  successful  mission 
operations 
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The  relationship  between  the  variables  is  shown  in  Figure  7.  As  the 
independent  (design)  variables  are  changed  in  accordance  with  (IAW)  a  pre¬ 
determined  interval,  we  are  interested  in  observing  the  change  in  the  dependent 
(criteria  space)  variables  and  correlating  this  change  with  the  strength  of  the  ties 
within  the  communications  network  (intervening  variables). 


Intervening  Variables 


Figure  7.  Parameter  Space  Variable  Relationship 

The  independent  variables — budget,  time,  and  external  threats — will  be  varied 
throughout  the  experiment  at  key  points,  and  the  dependent  variables  of  cost, 
schedule,  and  performance  will  be  measured  in  relation  to  the  variation  in  the  design 
variables.  Additionally,  we  will  observe  the  variation  and  link  strength  between  nodes 
in  the  intervening  variables  and  compare  them  to  the  criteria  space  as  well  as 
analyze  them  via  a  Pareto  analysis  method  to  determine  which  intervening  variables 
have  the  most  significant  impact  on  the  criteria  space.  Pareto  analysis  is  a  technique 
used  for  decision-making  based  on  the  Pareto  principle  and  known  as  the  80/20 
rule.  It  is  a  technique  that  statistically  separates  a  limited  number  of  input  factors  as 
having  the  greatest  impact  on  an  outcome,  either  desirable  or  undesirable.  Pareto 
analysis  is  based  on  the  idea  that  80%  of  a  project’s  benefit  can  be  achieved  by 
doing  20%  of  the  work,  or  conversely,  that  80%  of  problems  are  traced  to  20%  of  the 
causes. 

Pareto  Set  Relationship 

While  this  experiment  is  concerned  with  the  impact  of  the  independent 
variable  in  the  criteria  space,  we  are  also  focused  on  the  relationship  of  the 
intervening  variables  with  the  criteria  space  and  how  these  network  ties  tend  to 
influence  the  variation  in  cost,  schedule,  or  performance  of  the  program.  There  are 
10  attributes  in  the  descriptive  model  for  this  experiment.  Initially,  we  will  only  allow 
the  informal  and  formal  communications  pathways  to  vary,  and  we  will  measure  this 
variation  with  regard  to  the  strength  of  their  links  in  the  form  of  expressive, 
instrumental,  and  structural  strengths  and  weaknesses,  as  defined  in  Table  1 . 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  12- 


Using  the  Pareto  analysis  methodology,  this  will  reveal  which  of  these  links  has  the 
greatest  impact  on  the  dependent  variables.  Figure  8  reflects  how  20%  of  the 
intervening  variables  might  result  in  at  least  80%  impact  on  the  criteria  space. 


Link  Time 
(Strength) 


Communications  Network  Link  Type 


Figure  8.  Pareto  Chart  Showing  the  Relationship  of  Intervening  Variables 

The  variance  of  the  criteria  space  is  presented  in  terms  of  an  earned  value 
calculation,  which  allows  us  to  determine  the  level  of  expected  performance  value 
achieved  based  upon  the  cost  and  schedule  of  the  program.  Table  4  provides  a 
definition  of  key  earned  value  terms. 


Table  4.  Earned  Value  Management  Key  Terms 


Acronym 

Term 

Meaning 

BCWS 

Budgeted  Cost  for  Work  Scheduled 

Plan  -  Baseline  -  PMB 

BCWP 

Budgeted  Cost  for  Work  Performed 

Earned  Value 

ACWP 

Actual  Cost  of  Work  Performed 

Actuals 

BAC 

Budget  At  Completion 

Planned  Cost 

EAC 

Estimate  At  Completion 

Forecasted  Cost 

SV 

Schedule  Variance 

Accomplishment 

Variance 

CV 

Cost  Variance 

Earned  Value 
vs  Actual  Cost 

VAC 

Variance  At  Completion 

Forecasted 
Overrun  /  Underrun 

Subsequent  experiments  will  examine  additional  intervening  variables 
identified  in  the  descriptive  model.  While  these  variables  will  likely  have  an  impact  on 
the  criteria  space,  the  most  complex  set  of  intervening  variables  lies  within  the 
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formal  and  informal  communications  networks.  By  limiting  our  initial  experimental 
approach  to  just  the  communications  network  variables  that  lead  to  the  program 
decision  strategy,  we  are  reducing  the  overall  complexity  of  the  experiment  and 
improving  the  overall  validity  of  the  results. 

Conclusions 

This  experimental  design  paper  describes  the  discovery  experiment  phase  of 
an  overall  experimental  process  that  is  intended  to  provide  greater  insight  into  the 
root  cause  of  defense  program  execution  challenges.  This  experiment  will  set  the 
conditions  for  follow-on  hypothesis  testing  that  will  examine  the  effects  of  internal 
communications  networking  relationships  and  their  impact  on  the  value  of  a  program 
as  defined  by  cost,  schedule,  and  performance. 

If  characteristic  patterns  emerge  and  can  be  measured  and  predicted  based 
upon  the  variation  of  a  program  from  its  initial  conditions,  then  the  potential  exists  for 
identifying  policy,  organizational,  and  training  changes  from  existing  methods  that 
could  have  favorable  impacts  on  program  outcomes. 

Developing  an  experiment  to  examine  participant  performance  and  decision¬ 
making  patterns  and  impacts  in  a  controlled  program  scenario  could  provide  data  for 
the  continuous  refinement  of  predictive  analytical  decision  patterns  within  a  complex 
program.  Additionally,  the  experimental  process  can  provide  a  robust  training 
environment  for  future  program  teams  to  prepare  from  the  complex  and  ambiguous 
program  environments  prior  to  actually  taking  over  a  major  defense  acquisition 
program. 
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